Electronic device, control method thereof, and recording medium

ABSTRACT

In an operation by line-of-sight input, a position (designated position) designated by a gaze point is easier to view after a determination operation is performed by a user than before the determination operation is performed. An electric device includes a detector configured to detect a gaze point on a display unit, and a control unit configured to display an indicator indicating the gaze point on the display unit, wherein, in a case where a setting to keep displaying the indicator even in a case where an operation is performed on an operation unit by the user is set, the control unit performs control, in response to an operation performed on the operation unit by the user, to display a display item at the gaze point and to display the indicator and the display item in a changed display order.

BACKGROUND Field

The present disclosure relates to an electronic device that isconfigured to detect a line-of-sight of a user and is operable byline-of-sight input, a method of controlling the electronic device, anda recording medium.

Description of the Related Art

In recent years, an electronic device that is operated based on aline-of-sight of a user (hereinafter, “line-of-sight input”) is known.The line-of-sight input is effective especially in a case where the userwishes to input, to an electronic device (such as a digital camera or agame machine), an instruction to designate or move a desired positionquickly. Japanese Patent Application Laid-Open No. 9-114846 discusses atechnique by which an image similar to an image displayed at a gazepoint calculated from a line-of-sight of a user is prioritized indisplay order. According to Japanese Patent Application Laid-Open No.9-114846, if dedicated software is installed, a pointer can be displayedat a gaze point calculated from a line-of-sight of a user, and thus theuser can view the gaze point of the user and a movement of the gazepoint.

Since the technique discussed in Japanese Patent Application Laid-OpenNo. 9-114846 does not display the gaze point calculated from theline-of-sight of the user, it is not possible to check whether the gazepoint corresponds to a position that the user is viewing. In a casewhere the gaze point does not correspond to the position that the useris viewing, similar images that are not intended by the user may beprioritized. According to Tobii Technology AB, “Streaming”, Tobii GhostSoftware, <https://gaming tobii.com/software/ghost/> (searched theInternet on Oct. 7, 2020), a pointer indicating a gaze point isdisplayed. However, since the pointer is always displayed over anydisplayed objects, it is difficult to view a target displayed object,and consequently the user may feel annoyed with the pointer.

SUMMARY

The present disclosure is directed to a technique by which in performingan operation by line-of-sight input, a position (designated position)designated by a gaze point is easier to view after a determinationoperation is performed by a user than before the determination operationis performed.

According to various embodiments of the present disclosure, anelectronic device includes a detection unit configured to detect a gazepoint of a user viewing on a display unit, and at least one memory andat least one processor which function as a control unit configured todisplay an indicator indicating the gaze point on the display unit,wherein in a case where a setting to keep displaying the indicator evenin a case where an operation is performed on an operation unit by theuser is set, the control unit performs control, in response to anoperation performed on the operation unit by the user, to display adisplay item at the gaze point based on line-of-sight of the user and todisplay the indicator and the display item in a changed display order.

Further features of the present disclosure will become apparent from thefollowing description of example embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are external views illustrating a digital cameraaccording to an example embodiment of the present disclosure.

FIG. 2 is a block diagram illustrating a structure of a digital cameraaccording to an example embodiment of the present disclosure.

FIG. 3 is a flowchart illustrating a process of controlling a digitalcamera according to an example embodiment of the present disclosure.

FIGS. 4A1 to 4A3 and 4B1 to 4B3 are flowcharts illustrating a process ofcontrolling display of a line-of-sight pointer indicating a gaze pointof a user and a frame in imaging mode processing according to an exampleembodiment of the present disclosure.

FIGS. 5A and 5B are setting menu screens according to an exampleembodiment of the present disclosure.

FIGS. 6A to 6J are diagrams each illustrating a display example of aline-of-sight pointer indicating a gaze point of a user and a frameaccording to an example embodiment of the present disclosure.

FIGS. 7A and 7B are diagrams illustrating a display order of objectsdisplayed on a display unit according to an example embodiment of thepresent disclosure.

FIG. 8 is an external view illustrating a personal computer (PC)according to an example embodiment of the present disclosure.

FIGS. 9A to 9D are diagrams illustrating a display example in a casewhere an example embodiment of the present disclosure is applied to aPC.

DESCRIPTION OF THE EMBODIMENTS

Various example embodiments of the present disclosure will be describedin detail below with reference to the accompanying drawings.

It is to be noted that the following example embodiments are merelyexamples of implementations of the present invention and embodiments ofthe present disclosure can be appropriately modified or changeddepending on individual constructions and various conditions ofapparatuses to which the present invention is applied. Thus, the presentinvention is in no way limited to the following example embodiments.

<External View of Digital Camera 100>

Various example embodiments of the present disclosure will be describedbelow with reference to the drawings.

FIGS. 1A and 1B are external views illustrating a digital camera 100 asan example of an apparatus to which the present disclosure isapplicable. FIG. 1A is a front perspective view illustrating the digitalcamera 100, and FIG. 1B is a rear perspective view illustrating thedigital camera 100. In FIGS. 1A and 1B, a display unit 28 is a displayunit that displays an image and various types of information and isprovided to a rear surface of the digital camera 100. A touch panel 70 ais an operation member on which a touch operation can be performed. Thetouch panel 70 a detects a touch operation on a display surface(operation surface) of the display unit 28. An outside-finder displayunit 43 is a display unit that is provided outside a finder of thedigital camera 100 and displays various setting values of the digitalcamera 100, such as a shutter speed and an aperture.

A shutter button 61 is an operation unit for issuing an imaginginstruction. A mode selection switch 60 is an operation unit forselecting various modes. A terminal cover 40 is a cover that protects aconnector (not illustrated) for connecting a connection cable of anexternal device to the digital camera 100. A main electronic dial 71 isa rotary operation member of an operation unit 70, and setting values,such as the shutter speed and the aperture, are changed by rotating themain electronic dial 71. A power switch 72 is an operation member forturning on/off the digital camera 100. A sub-electronic dial 73 is arotary operation member of the operation unit 70 for moving a selectedframe and forwarding an image. A four-direction key 74 of the operationunit 70 is an operation member including a button that can be pressed infour directions, and an operation is performed based on a direction inwhich the four-direction key 74 is pressed. A SET button 75 is a buttonof the operation unit 70 and is used mainly to determine a selecteditem. A moving image button 76 is used to issue an instruction tostart/stop imaging (recording) of a moving image. An auto-focus-on(AF-ON) button 77 of the operation unit 70 is pressed to start anauto-focus (AF) operation (automatic focal point adjustment). While theAF operation is started mainly in response to a press of the shutterbutton 61, an instruction to start an AF operation can be issued also bypressing the AF-ON button 77. In the digital camera 100 that can be setto not perform an AF operation in response to a press of the shutterbutton 61, an AF start instruction and an imaging instruction can beseparately issued. By pressing the AF-ON button 77 and then pressing theshutter button 61, imaging is performed with the AF position fixed, orimaging is performed even under a situation where AF is inoperable. Anauto-exposure (AE) lock button 78 of the operation unit 70 is pressed inan imaging standby state (state where image capturing is started in animaging mode using an image capturing unit 22, preparation for imagingis not performed, and the digital camera 100 is waiting in the imagingmode), to fix an exposure state. Specifically, imaging is performedusing a fixed user-desired exposure value. A reproduction button 79 ofthe operation unit 70 is an operation button for changing between theimaging mode and a reproduction mode. The reproduction button 79 ispressed during the imaging mode to change to the reproduction mode, andthe display unit 28 displays the latest image among images recorded on arecording medium 200. A focusing button 84 of the operation unit 70 ispressed to focus a lens at an aperture value that is set at the time ofthe press of the focusing button 84 to check an in-focus range (depth offield). A menu button 81 of the operation unit 70 is pressed to displaya menu screen to be used for various settings on the display unit 28. Anenlargement button 82 of the operation unit 70 is an operation buttonfor enabling or disabling an enlargement mode during a live view displayin the imaging mode. A live-view image is enlarged or reduced byenabling the enlargement mode and then operating the main electronicdial 71. In the reproduction mode, the enlargement button 82 functionsas an enlargement button that enlarges a reproduction image andincreases an enlargement ratio. A multi-controller 83 is an operationmember including a direction key and a button. The direction key can beoperated in eight directions, and the button can be pressed. Anoperation based on a corresponding direction in which themulti-controller 83 is pushed is performed. A user can intuitively setvarious settings on the menu screen displayed on the display unit 28using the four-direction key 74, the SET button 75, and themulti-controller 83. A framing assist button (not illustrated) is anoperation button provided to a lens unit 150 or a lens adapterconnecting the digital camera 100 to the lens unit 150. The framingassist button is an operation member disposed at a position where theuser can press the framing assist button with the left hand of the userwhen holding a grip portion 90 with the right hand of the user andholding the lens unit 150 with the left hand of the user during imaging.

The operation unit 70 includes various operation members as an inputunit for receiving user operations. The operation unit 70 includes apush button, a rotary dial, and a touch sensor and at least includes theshutter button 61, the touch panel 70 a, the main electronic dial 71,the power switch 72, the sub-electronic dial 73, the four-direction key74, the SET button 75, the moving image button 76, the AF-ON button 77,the AE lock button 78, the reproduction button 79, the menu button 81,the enlargement button 82, the multi-controller 83, the focusing button84, and the framing assist button. A line-of-sight determinationfunction is a function of updating an AF frame to a position based on agaze point, and the digital camera 100 can include a dedicated buttonhaving the line-of-sight determination function, or the line-of-sightdetermination function can be assigned to an operation member havinganother function. Operation members to which the line-of-sightdetermination function can be assigned are the AF-ON button 77, the AElock button 78, the push button of the multi-controller 83, the focusingbutton 84, and the framing assist button. The AF-ON button 77, the AElock button 78, and the multi-controller 83 are disposed at a positionoperable by the user without disrupting imaging while the user operatesthe shutter button 61 with the forefinger of the right hand of the user.Specifically, when the user holds the grip portion 90 with the righthand of the user, the AF-ON button 77, the AE lock button 78, and themulti-controller 83 are on an upper side (side closer to the shutterbutton 61) of a central position on a rear surface portion of thedigital camera 100, which is on the opposite side to the subject, and onthe right hand side (side closer to the grip portion 90) of an electricviewfinder (EVF) 29. Operation members to which the line-of-sightdetermination function can be assigned are not limited to those providedto the rear surface portion of the digital camera 100 and can be anyoperation member that the user can operate while operating the shutterbutton 61. For example, the line-of-sight determination function can beassigned to an operation member that is provided to a front surfaceportion (subject side) or to the lens unit 150, such as the focusingbutton 84 or the framing assist button. In this case, the position ofthe operation member is not limited to the rear surface of the digitalcamera 100 and can be any position at which the operation member can beoperated by the user with a finger other than the forefinger of theright hand operating the shutter button 61. An operation member to whichthe user can assign the line-of-sight determination function and thathas another function is a button having a function not causing thedigital camera 100 to change from the imaging mode when operated whilethe shutter button 61 is operated or a button having a function thatdoes not disturb execution of an imaging function by an operation on theshutter button 61. Further, a button to which various functions can beassigned and that can be pressed can be employed. Alternatively, thefunction can be triggered in response to an operation performed on anoperation bar operable rightward and leftward or a rotatable ring, orthe touch panel 70 a capable of detecting pressing force describedbelow, instead of a button, is pressed with great pressure.

A communication terminal 10 is a communication terminal forcommunication of the digital camera 100 with the lens unit 150(removable) described below. An eyepiece 16 is an eyepiece of a finder(view-through type finder), and the user can view a video imagedisplayed on the EVF 29 of an inside-finder display unit through theeyepiece 16. An eye approach detection unit 57 is an eye detectionsensor that detects whether an eye of a user capturing an image is putto the eyepiece 16. A cover 202 is a cover for a slot in which therecording medium 200 is stored. The grip portion 90 is a holding unithaving a shape that is easy for the right hand of the user to hold whenthe user holds the digital camera 100. The shutter button 61 and themain electronic dial 71 are disposed at a position easily operable bythe forefinger of the right hand of the user in a state where the userholds the digital camera 100 by holding the grip portion 90 with thelittle finger, the ring finger, and the middle finger of the right hand.Further, the sub-electronic dial 73 is disposed at a position operableby the thumb of the right hand in the foregoing state.

FIG. 2 is a block diagram illustrating an example of a configuration ofthe digital camera 100 according to the present example embodiment. InFIG. 2 , the lens unit 150 is a lens unit on which an interchangeableimaging lens is mounted. A lens 103 normally consists of a plurality oflenses but is illustrated simply as a single lens in FIG. 2 . Acommunication terminal 6 is a communication terminal for communicationof the lens unit 150 with the digital camera 100. The lens unit 150communicates with a system control unit 50 via the communicationterminal 6 and the communication terminal 10, and a lens system controlcircuit 4 in the lens unit 150 controls a diaphragm 1 via a diaphragmdriving circuit 2. Then, the lens unit 150 adjusts a focal point of thelens 103 by displacing the lens 103 via an AF driving circuit 3.

A shutter 101 is a focal plane shutter that freely controls an exposuretime of the image capturing unit 22 under control by the system controlunit 50.

The image capturing unit 22 is an image sensor including acharge-coupled device (CCD) sensor or a complementary metal oxidesemiconductor (CMOS) sensor and converts an optical image into anelectric signal. An analog/digital (A/D) conversion device 23 is used toconvert an analog signal output from the image capturing unit 22 into adigital signal.

An image processing unit 24 performs predetermined pixel interpolation,resizing processing, such as reduction, and color conversion processingon data from the A/D conversion device 23 or data from a memory controlunit 15 described below. Further, the image processing unit 24 performspredetermined calculation processing using captured image data. Based ona calculation result acquired from the image processing unit 24, thesystem control unit 50 controls exposure and distance measurement.Consequently, through-the-lens (TTL) AF processing, AE processing, andflash with pre-emission (EF) processing are performed. The imageprocessing unit 24 further performs predetermined calculation processingusing captured image data and performs TTL automatic white balance (AWB)processing using an obtained result of the calculation.

The memory control unit 15 controls data transmission and receptionbetween the A/D conversion device 23, the image processing unit 24, anda memory 32. Data output from the A/D conversion device 23 is written tothe memory 32 via the image processing unit 24 and the memory controlunit 15 or directly via the memory control unit 15. The memory 32 storesimage data that is acquired by the image capturing unit 22 and thenconverted into digital data by the A/D conversion device 23 and imagedata to be displayed on the display unit 28 or the EVF 29. The memory 32has sufficient capacity to store a predetermined number of still imagesand a predetermined time of moving images and audio.

The memory 32 is also used as a memory (video memory) for image display.Image data for display that is written to the memory 32 is displayed onthe display unit 28 or the EVF 29 via the memory control unit 15. Thedisplay unit 28 and the EVF 29 perform display on a liquid crystaldevice (LCD) or an organic electroluminescent (EL) display, based on asignal from the memory control unit 15. Data converted from analog datainto digital data by the A/D conversion device 23 and stored in thememory 32 is sequentially transferred to the display unit 28 or the EVF29 and displayed to conduct live view display (LV display). Hereinafter,an image displayed in live view will be referred to as a live view image(LV image).

An infrared light emitting diode 166 is a light emitting element fordetecting a line-of-sight of the user on a screen in the finder andemits infrared light to an eyeball (eye) 161 of the user. The infraredlight emitted from the infrared light emitting diode 166 is reflected bythe eyeball (eye) 161, and the reflected infrared light travels to adichroic mirror 162. The dichroic mirror 162 reflects only infraredlight and transmits visible light. The reflected infrared light havingits optical path changed passes through an imaging lens 163 and forms animage on an image capturing surface of a line-of-sight detection sensor164. The imaging lens 163 is an optical member of a line-of-sightdetection optical system. The line-of-sight detection sensor 164includes an image sensor, such as a CCD image sensor.

The line-of-sight detection sensor 164 photoelectrically convertsincident reflected infrared light into an electric signal and outputsthe electric signal to a line-of-sight detection circuit 165. Theline-of-sight detection circuit 165 includes at least one processor. Theline-of-sight detection circuit 165 detects the line-of-sight of theuser from an image or movement of the eyeball (eye) 161 of the user,based on a signal output from the line-of-sight detection sensor 164 andoutputs detected information to the system control unit 50. The dichroicmirror 162, the imaging lens 163, the line-of-sight detection sensor164, the infrared light emitting diode 166, and the line-of-sightdetection circuit 165 form a line-of-sight detection block 160. Theline-of-sight detection block 160 is one of reception units that receiveline-of-sight input.

In various embodiments of the present disclosure, a line-of-sight isdetected by a corneal reflection method using the line-of-sightdetection block 160. The corneal reflection method is a method thatdetects a direction of a line-of-sight of the user by detecting movementof the eye based on a relationship between infrared light emitted fromthe infrared light emitting diode 166 and reflected from the eyeball(eye) 161, especially the cornea, and a position of the pupil of theeyeball (eye) 161. There are various other line-of-sight detectionmethods, such as a scleral reflection method using a difference in lightreflectance between black and white eyes. Any line-of-sight detectionmethods that can detect a line-of-sight other than those described abovecan also be employed.

The outside-finder display unit 43 displays various setting values ofthe digital camera 100 including a shutter speed and an aperture via anoutside-finder display unit driving circuit 44.

A non-volatile memory 56 is an electrically erasable and programmablememory and uses, for example, a flash read-only memory (flash-ROM). Thenon-volatile memory 56 stores constant numbers for operation of thesystem control unit 50 and programs. The programs herein are programsfor executing various flowcharts described below according to thepresent example embodiment.

The system control unit 50 is a control unit including at least oneprocessor or circuit and controls the entire digital camera 100. Thesystem control unit 50 executes the programs stored in the non-volatilememory 56 to realize below-described processing according to the presentexample embodiment. A system memory 52 is, for example, a random accessmemory (RAM), and the constant numbers and variable numbers foroperation of the system control unit 50 and programs read from thenon-volatile memory 56 are developed onto the system memory 52. Further,the system control unit 50 controls display by controlling the memory 32and the display unit 28.

A system timer 53 is a time measurement unit that measures a time foruse in various types of control and the time of a built-in clock.

The mode selection switch 60 is an operation member of the operationunit 70 and switches an operation mode of the system control unit 50between a still image capturing mode and a moving image capturing mode.The still image capturing mode includes an automatic imaging mode, anautomatic scene determination mode, a manual mode, an aperture prioritymode (Av mode), a shutter speed priority mode (Tv mode), and a programAE mode (P mode). The still image capturing mode further includesvarious scene modes with different imaging settings each correspondingto a different imaging scene, a custom mode, and the like. The user canchange the operation mode directly to any of the foregoing modes usingthe mode selection switch 60. Alternatively, the user uses the modeselection switch 60 to change to an imaging mode list screen, selects amode from a plurality of displayed modes, and then uses anotheroperation member to change the operation mode to the selected mode.Similarly, the moving image capturing mode can include a plurality ofmodes.

A first shutter switch 62 is turned on when the shutter button 61 of thedigital camera 100 is operated halfway, i.e., half-pressed (imagingpreparation instruction), and the first shutter switch 62 generates afirst shutter switch signal SW1. Imaging preparation operations, such asAF processing, AE processing, AWB processing, and EF processing arestarted based on the first shutter switch signal SW1.

A second shutter switch 64 is turned on when the shutter button 61 isoperated completely, i.e., fully pressed (imaging instruction), and thesecond shutter switch 64 generates a second shutter switch signal SW2.The system control unit 50 starts a series of image processing fromreading of a signal from the image capturing unit 22 to writing of acaptured image as an image file to the recording medium 200, based onthe second shutter switch signal SW2. In a case where the second shutterswitch 64 is continuously on, the digital camera 100 continuouslycaptures an image (performs continuous imaging) at a speed that dependson a predetermined continuous imaging speed.

A power control unit 80 includes a battery detection circuit, a directcurrent (DC)-DC converter, and a switch circuit for changing a block towhich a current is to be applied. The power control unit 80 detectsattachment of a battery, a battery type, and a battery level. Further,the power control unit 80 controls the DC-DC converter, based on aresult of the detection and an instruction from the system control unit50 and feeds a predetermined voltage for a predetermined time to eachcomponent including the recording medium 200. A power unit 30 includes aprimary battery, such as an alkali battery and a lithium battery, asecond battery, such as a nickel-cadmium (NiCd) battery, a nickel metalhydride (NiMH) battery, and a lithium (Li) battery, and an alternatingcurrent (AC) adapter.

A recording medium interface (recording medium I/F) 18 is an interfacewith the recording medium 200, such as a memory card or a hard disk. Therecording medium 200 is a recording medium, such as a memory card, thatrecords captured images and includes a semiconductor memory or amagnetic disk.

A communication unit 54 is connected via a wireless or wired cable andtransmits and receives video signals and audio signals. Thecommunication unit 54 can also be connected to a wireless local areanetwork (wireless LAN) and the Internet. Further, the communication unit54 can communicate with an external device via Bluetooth® or Bluetooth®Low Energy. The communication unit 54 can transmit an image (including alive view image) captured by the image capturing unit 22 and an imagerecorded in the recording medium 200 and can receive images and varioustypes of information from external devices.

An orientation detection unit 55 detects an orientation of the digitalcamera 100 with respect to the gravity direction. Whether an imagecaptured by the image capturing unit 22 is an image captured with thedigital camera 100 held horizontally or vertically is determined basedon the orientation detected by the orientation detection unit 55. Thesystem control unit 50 can add direction information based on theorientation detected by the orientation detection unit 55 to an imagefile of an image captured by the image capturing unit 22 and can rotatethe image to record the rotated image. An acceleration sensor or a gyrosensor can be used as the orientation detection unit 55. A movement ofthe digital camera 100 (pan, tilt, lift, still state) can be detectedusing the acceleration sensor or the gyro sensor as the orientationdetection unit 55.

The eye approach detection unit 57 is an eye proximity detection sensorthat detects (approach detection) an approach (eye approach) orseparation (eye separation) of the eye (object) 161 to or from theeyepiece 16 of the finder. The system control unit 50 switches thedisplay unit 28 and the EVF 29 between displaying (display state) andnon-displaying (non-display state), based on a state detected by the eyeapproach detection unit 57. More specifically, in a case where at leastthe digital camera 100 is in the imaging standby state and a displaydestination selection setting for a live view image captured by theimage capturing unit 22 is set to an automatic selection setting, andwhen an eye approach is not detected, the display unit 28 is set as adisplay destination and the display on the display unit 28 is turned onwhile the display on the EVF 29 is set to a non-display state. When aneye approach is detected, the EVF 29 is set as a display destination,and the display on the EVF 29 is turned on while the display on thedisplay unit 28 is set to a non-display state. The eye approachdetection unit 57 can use, for example, an infrared ray proximity sensorand detects an object approach to the eyepiece 16 of the finderincluding the EVF 29 therein. In a case where an object approaches, aninfrared ray projected from a light projecting unit (not illustrated) ofthe eye approach detection unit 57 is reflected, and a reflectedinfrared ray is received by a light receiving unit (not illustrated) ofthe infrared ray proximity sensor. Based on the amount of receivedinfrared ray, a distance of the approaching object from the eyepiece 16(approaching eye distance) is also determined. The eye approachdetection unit 57 performs approaching eye detection to detect an objectapproaching the eyepiece 16 as described above. According to the presentexample embodiment, the light projecting unit and the light receivingunit of the eye approach detection unit 57 are devices disposedseparately from the infrared light emitting diode 166 and theline-of-sight detection sensor 164. Alternatively, the infrared lightemitting diode 166 can be also used as the light projecting unit of theeye approach detection unit 57. Further, the line-of-sight detectionsensor 164 can be also used as the light receiving unit. In a case wherean approaching object is detected within a predetermined distance fromthe eyepiece 16 in a state where the eye 161 is not in the proximity(non-approach state), it is determined that the eye 161 is in theproximity. In a case where a detected approaching object is separated bya predetermined distance or longer in the state where the eye 161 is inthe proximity (approach state), it is determined that the eye 161 isseparated. A threshold value for eye approach detection and a thresholdvalue for eye separation detection can be set differently by, forexample, providing hysteresis. Further, after an eye approach isdetected, the approach state continues until an eye separation isdetected. After an eye separation is detected, the non-approach statecontinues until an eye approach is detected. The infrared ray proximitysensor is a mere example, and a different sensor that detects anapproach of an eye or object as an eye approach can be employed as theeye approach detection unit 57.

The system control unit 50 performs one of the operations or statesdescribed below based on output from the line-of-sight detection block160.

A state where a line-of-sight of a user having an eye approaching theeyepiece 16 is detected by the line-of-sight detection block 160, i.e.,a state where a line-of-sight input is received.

A state where a line-of-sight of a user having an eye approaching theeyepiece 16 is detected by the line-of-sight detection block 160 and theuser is gazing (described below).

A state where a line-of-sight of a user having an eye approaching theeyepiece 16 is detected by the line-of-sight detection block 160 andthereafter the line-of-sight is no longer detected, i.e., a state wherea line-of-sight input ends.

A state where a line-of-sight of a user having an eye approaching theeyepiece 16 is not detected by the line-of-sight detection block 160,i.e., a state where a line-of-sight input is not received.

The state where “the user is gazing” refers to a state where theline-of-sight detection block 160 identifies a position (gaze point) onthe EVF 29, based on a detected line-of-sight and determines that theamount of movement of the identified gaze point does not exceed apredetermined amount within a predetermined time. Specifically, the gazepoint refers to a position on the EVF 29 (on a display unit) that isdetermined as being viewed by the user.

The state where “a line-of-sight input ends” is a case where the eye 161of the user is separated from the eyepiece 16 or a case where the eye161 remains in the proximity of the eyepiece 16 but is not detectedbecause the eyelid is closed.

The touch panel 70 a and the display unit 28 can be integrated together.For example, the touch panel 70 a is configured in such a manner thatthe light transmittance does not disturb the display on the display unit28, and the touch panel 70 a is attached to an upper layer of a displaysurface of the display unit 28. Input coordinates on the touch panel 70a are associated with display coordinates on a display screen of thedisplay unit 28. This provides a display object (graphical userinterface) that looks as though the user can directly operate the screendisplayed on the display unit 28. The system control unit 50 detects oneof the operations described below on the touch panel 70 a or one of thestates described below.

A finger or a pen that is previously not in touch with the touch panel70 a touches the touch panel 70 a. Specifically, a touch starts(hereinafter, referred to as “touch-down”).

A state where the touch panel 70 a is being touched with a finger or apen (hereinafter, referred to as “touch-on”).

A state where a finger or a pen touching the touch panel 70 a is moved(hereinafter, referred to as “touch-move”).

A finger or a pen touching the touch panel 70 a is separated.Specifically, the touch ends (hereinafter, referred to as “touch-up”).

A state where the touch panel 70 a is not touched (hereinafter, referredto as “touch-off”).

In a case where a touch-down is detected, a touch-on is simultaneouslydetected. After the touch-down, unless a touch-up is detected, atouch-on is normally detected continuously. A state where a touch-moveis detected is also a state where a touch-on is detected. Even if atouch-on is detected, unless the touch position is not moved, atouch-move is not detected. After a touch-up of all the touched fingersor a pen is detected, the state is changed to a touch-off.

The foregoing operations or states and coordinates of a position touchedwith a finger or a pen on the touch panel 70 a are notified to thesystem control unit 50 via an internal bus. The system control unit 50determines which operation (touch operation) is performed on the touchpanel 70 a, based on the notified information. As to a touch-move, amovement direction of a finger or a pen moving on the touch panel 70 acan be determined for each vertical component and each horizontalcomponent on the touch panel 70 a, based on a change in the positioncoordinates. In a case where a touch-move of a predetermined distance orgreater is detected, it is determined that a slide operation isperformed. An operation of suddenly moving a finger touching the touchpanel 70 a by a distance and then releasing the finger is referred to as“flick”. In other words, a flick is an operation of quickly sliding afinger on the touch panel 70 a in a similar way to flipping the surfaceof the touch panel 70 a. In a case where a touch-move of a predetermineddistance or greater at a predetermined speed or higher is detectedfollowed by detection of a touch-up, it is determined that a flick isperformed (it is determined that a flick is performed following a slideoperation). Further, a touch operation of simultaneously touching aplurality of positions (e.g., two points) and bringing the touchedpositions closer to each other will be referred to as “pinch-in” and atouch operation of bringing the touched positions away from each otheras “pinch-out”. The pinch-out and the pinch-in will collectively bereferred to as “pinch operation” (or simply “pinch”). The touch panel 70a can be a touch panel of any of various methods, such as a resistivefilm method, a capacitance method, a surface acoustic wave method, aninfrared method, an electromagnetic induction method, an imagerecognition method, and an optical sensor method. Some methods detect atouch based on a touch on a touch panel while some other methods detecta touch based on an approach of a finger or a pen to a touch panel, butany of the methods can be employed.

Settings such as those that are illustrated as items 501 and 502 in FIG.5A are set as AF settings. One of “single-point AF” and “entire-regionAF” is set as an AF area (AF frame setting method) setting asillustrated in the item 501. The single-point AF setting indicates thatan AF execution target position in a LV image is a single point (oneposition). The user can set a single point as an AF execution targetposition, and AF is executed at the display position of a single-pointAF frame that is displayed based on a user instruction to execute AF.For example, in imaging a train running toward the user, the user maywish to execute AF on an operator seat of the train and to capture animage of the train with a composition including the entire train. Inthis case, imaging is performed with the AF frame fixed at the singlepoint (one position) at the moment at which the operator seat of thetrain reaches the position, whereby an image focused on the user-desiredposition with the user-desired composition is acquired. On the otherhand, the entire-region AF setting illustrated in the item 502 is asetting that enables AF execution (enables focal point detection) on anentire AF executable region of a region displayed on the EVF 29 of thedigital camera 100. In imaging a subject with unpredictable movement,the AF area setting is set to the entire region, to execute AF on asubject regardless of where the subject is in the AF executable region.

One of tracking settings “ON” and “OFF” is set as an AF operationsetting as illustrated in the item 502 in FIG. 5A. The tracking functionset to “ON” is suitable for imaging in a situation where a trackingtarget is a subject with a continuously-changing (moving) imagingdistance, such as an athlete in competition, a moving child, or ananimal. Tracking and continuous AF are executed on an AF target subjectdetermined using an element condition. The continuous AF refers tocontinuous execution of AF on a tracking target subject.

The tracking function set to “OFF” is suitable for imaging of a stillsubject, and an AF frame position is determined based on an elementcondition (priority) described below at the time when the first shutterswitch 62 is on. At this time, the subject is not tracked.

In a case where the tracking function is set to “ON”, if a face of aperson is detected from a LV image, the face is preferentiallydetermined as an AF target subject. In a case where a plurality of facesof persons is detected, one face is selected based on one of the belowpredetermined priority levels (conditions), for example, and is set asan AF target subject.

The face is a face of the detected person.

The size of the face is large.

The position of the face is close to the digital camera 100 (nearerside).

The position of the face is close to the center of the image.

The face is a face of a pre-registered person.

In a case where a face of a person is not detected, a subject other thana face is selected based on one of the following predetermined prioritylevels (conditions), for example, and set as an AF target subject.

A subject close to the digital camera 100 (nearer side).

A subject with a high contrast.

A subject with high priority, such as an animal or a vehicle.

A subject that is a moving object.

In a case where a tracking target subject is designated by the user, thetracking target subject is set as an AF target subject. Specifically,weighting is conducted using at least one element condition among theelement conditions described above as priority examples of the twogroups, and a subject with an obtained score higher than or equal to apredetermined threshold or a subject with the highest score isdetermined as an AF target subject.

There are four patterns of combinations of settings of the items 501 and502 that can be set as the AF settings. The user uses different settingsfor different imaging target subjects or imaging situations, to setoptimum imaging settings for the user, whereby imaging is performed at adesired timing with a desired composition.

In FIG. 5A, the AF area is set to “single-point” AF, and the tracking isset to “OFF”. This indicates that the user wishes to fix the AF frame toa single point. This combination of settings often improves useroperability in, for example, imaging a still subject, such as a flower.

FIG. 5A illustrates an example of a setting menu screen. Whether todetect a line-of-sight of the user, i.e., whether to drive theline-of-sight detection block 160, is selected by switching aline-of-sight input function of an item 503 between “ON” and “OFF”.Specifically, whether to receive an operation by line-of-sight input isset by switching the line-of-sight input function of the item 503between “ON” and “OFF”.

In a case where the user selects an item 504, the screen is changed to ascreen (screen of the bottom layer of the setting menu screen in FIG.5A) as illustrated in FIG. 5B. FIG. 5B illustrates a screen on whichdetailed settings of a line-of-sight AF function can be set. Theline-of-sight AF function of an item 505 is a function of executing AFto focus on a subject displayed at a gaze point. In a case where theitem 505 is set to “ON”, a subject is identified based on a movement ofa line-of-sight of the user, and AF is executed on an identifiedsubject, and accordingly the user can intuitively execute AF on adesired subject. An item 506 is a setting item about whether to displaya line-of-sight pointer at a gaze point. In a case where the item 506 isset to “ON”, the line-of-sight pointer is constantly displayed at a gazepoint detected by the line-of-sight detection block 160, based on thedirection in which the eye of the user is gazing. On the other hand, ina case where the item 506 is set to “OFF”, the line-of-sight pointer isnot displayed constantly even in a case where a gaze point is detectedby the line-of-sight detection block 160. With this configuration, theuser can use the line-of-sight pointer to recognize the gaze point ofthe user.

The configuration of the item 506 is not limited to the above-describedsetting, i.e., the setting simply about whether to display or notdisplay (ON/OFF) the line-of-sight pointer at the gaze point.Alternatively, the user can select from any one of three choices ofsettings that are “constantly display the line-of-sight pointer”,“display the line-of-sight pointer only during imaging standby”, and “donot display the line-of-sight pointer”. With the three choices, thedisplay control that satisfies needs of the user who wishes to usedisplay of the line-of-sight pointer to check the detected gaze point ofthe user but feels annoyed if the line-of-sight pointer is constantlydisplayed can be performed.

Specifically, the control is performed as follows. In the setting“constantly display the line-of-sight pointer”, in a case where theline-of-sight input function is “ON” and a gaze point is being detected,the line-of-sight pointer is displayed at the gaze point detected by theline-of-sight detection block 160.

In the setting “display the line-of-sight pointer only during imagingstandby”, the line-of-sight pointer is displayed at the gaze pointduring imaging standby, i.e., not during execution of AF in imaging modeprocessing. On the other hand, the displayed line-of-sight pointer ishidden during execution of AF in imaging mode processing. The period“during execution of AF” is, in other words, “the period of when aninstruction to perform AF is issued by the user”. Thus, for example, theline-of-sight pointer is hidden in a case where the first shutter switch62 is kept turned on, and the line-of-sight pointer is displayed (thedisplay of the line-of-sight pointer is resumed) in a case where thefirst shutter switch 62 is released from turned on state. Further, theline-of-sight pointer is hidden in a case where the AF-ON button 77 iskept turned on, and the line-of-sight pointer is displayed (the displayof the line-of-sight pointer is resumed) in a case where the AF-ONbutton 77 is released from turned on state.

Further, even in a case where the line-of-sight pointer display settingis set to the setting “constantly display the line-of-sight pointer”,there is also a case where switching displaying and non-displaying ofthe line-of-sight pointer based on whether AF is executed is desirable,depending on an AF operation setting. Two types of AF operations will bediscussed below as an example. Servo AF refers to an AF operation ofcontinuously performing AF on a subject in a case where the firstshutter switch 62 is kept turned on. One-shot AF refers to an AFoperation of performing AF once at a timing of when the first shutterswitch 62 is turned on, and the AF position determined by the turning onof the first shutter switch 62 remains unchanged even if the subject orthe digital camera 100 moves after the turning on. The servo AF is formoving subject image capturing, and the one-shot AF is for still subjectimage capturing. In the servo AF, since AF is continuously executed on asubject during the turning on of the first shutter switch 62, the usercan view an AF execution position with ease even in a case where theline-of-sight pointer is not hidden. Thus, in a case of the servo AF,the line-of-sight pointer is constantly displayed based on the setting“constantly display the line-of-sight pointer” regardless of whether AFis executed. On the other hand, in a case of the one-shot AF, the AFposition is determined only once at a timing of when the first shutterswitch 62 is turned on, and thus if display of the line-of-sight pointeris continued after the turning on, the eye of the user may be caught bythe line-of-sight pointer and the user may have difficulty in viewingthe AF execution position. Thus, in the one-shot AF, even in a casewhere the setting “constantly display the line-of-sight pointer” is set,the line-of-sight pointer is hidden during execution of AF, and theline-of-sight pointer is displayed not during execution of AF (duringimaging standby).

As described above, displaying and non-displaying of the line-of-sightpointer is switched as appropriate in each situation. Thus, the user canview the detected gaze point of the user as appropriate in eachsituation. Further, since the line-of-sight pointer is not constantlydisplayed, the user is less likely to feel annoyed by the line-of-sightpointer.

In the setting “do not display the line-of-sight pointer”, as the nameof the setting indicates, the line-of-sight pointer is not displayedeven in a case where the line-of-sight input function is “ON” and a gazepoint of the user is detected by the line-of-sight detection block 160.

According to the present example embodiment, the line-of-sight pointeris displayed as an indicator as illustrated as a pointer 606 in FIG. 6D.A small circle centered at the gaze point is displayed, and a largercircle than the small circle is displayed around the small circle. InFIG. 6D, the small circle of the pointer 606 is displayed with itsinside portion being shaded, whereas the larger circle is displayed withits frame being shaded. Each of the shaded portions is translucent toprevent the subject from becoming completely invisible when the shadedportions are superimposed over the subject or the AF frame in the LVimage. Further, similarly, since the region between the small circle andthe frame of the larger circle is not shaded, the visibility of the LVimage is not lost. A display form of the line-of-sight pointer is notlimited to the above described form. For example, the circle at thecenter of the pointer 606 in FIG. 6D can be changed to a crosshairs todisplay the gaze point more clearly to the user. Further, the displayform of the line-of-sight pointer is not limited to the double circles.Alternatively, the line-of-sight pointer can be displayed simply as asingle circle. A shape of the line-of-sight pointer can be changed bythe user as desired. Further, not only the shape but also a color and asize of the line-of-sight pointer can be changed. For example, the colorcan be selected from orange, purple, and white as desired, and the sizecan be selected from large, medium, and small as desired.

An item 507 is a setting item about whether to use an operation ofturning on the first shutter switch (SW1) 62, i.e., an operation ofhalf-pressing the shutter button 61, as an operation (line-of-sightdetermination operation) of determining a gaze point. In a case wherethe item 507 is enabled, a point detected by the line-of-sight detectionblock 160 when the shutter button 61 is half-pressed by the user isdetermined as a gaze point. In a case where the item 507 is disabled,half-pressing the shutter button 61 does not determine the gaze point.

For example, the line-of-sight determination function is assigned to theAF-ON button 77 at the time of initial setting (at the time of shipmentfrom factory). In a case where the item 507 is set to “enabled”, aline-of-sight determination operation is performed in a case where notthe AF-ON button 77 but the shutter button 61 is pressed to turn on thefirst shutter switch 62. Further, similarly, the line-of-sightdetermination operation can be assigned to a button of the operationunit 70 to which another function is assigned. A case where a gaze pointdetermined using another button is different from a gaze point at thehalf-press of the shutter button 61 in a case where the userhalf-presses the shutter button 61 after determining the gaze pointusing the button of another function will be discussed below as anexample. In this case, since the gaze point at the half-press of theshutter button 61 may be determined to cause AF execution on a subjectat this gaze point, which is different from the user-desired gaze pointdetermined using the button of the other function, operability isinferior. Thus, the item 507 on the setting menu screen is provided toallow the user to set whether to determine a gaze point by turning onthe first shutter switch (SW1) 62.

The control according to a first example embodiment will be describedbelow with reference to flowcharts illustrated in FIGS. 3 and 4A.According to the first example embodiment, in step S302 (imaging modeprocessing) in FIG. 3 , frames (items) such as the single-point AF frameand a subject frame that are displayed before a line-of-sightdetermination operation by the user are displayed at a gaze point as aresult of the line-of-sight determination operation.

FIG. 3 is a flowchart illustrating a control process by the digitalcamera 100 according to the present example embodiment. The controlprocess is realized by the system control unit 50 by developing aprogram stored in the non-volatile memory 56 onto the system memory 52and executing the developed program. The flowchart in FIG. 3 is startedin a case where the digital camera 100 is turned on (activated). Theflowchart in FIGS. 4A1 to 4A3 is started in a case where imaging modeprocessing is to be executed (YES in step S301) in the flowchart in FIG.3 . Further, FIGS. 6A to 6J are diagrams illustrating an example of adisplay on the EVF 29 in a case where the control process illustrated inthe flowchart in FIG. 3 is performed. FIGS. 6A to 61 are diagramsillustrating an example of a display in a case where the AF area settingof the item 501 in FIG. 5A is set to “single point” and the trackingfunction setting of the item 502 is set to “ON”. FIG. 6J is a diagramillustrating an example of a display in a case where the AF area settingis set to “entire region” and the tracking function is set to “OFF”.Further, the control in FIG. 3 is for a case where the line-of-sightpointer display setting is set to the setting “constantly display theline-of-sight pointer”.

In step S301, the system control unit 50 determines whether the imagingmode is set. In a case where the imaging mode is set (YES in step S301),the processing proceeds to step S302. In a case where the imaging modeis not set (NO in step S301), the processing proceeds to step S303.

In step S302, the system control unit 50 performs imaging modeprocessing. The imaging mode processing will be described below withreference to FIGS. 4A1 to 4A3.

In step S303, the system control unit 50 determines whether thereproduction mode is set. In a case where the reproduction mode is set(YES in step S303), the processing proceeds to step S304. In a casewhere the reproduction mode is not set (NO in step S303), the processingproceeds to step S305.

In step S304, the system control unit 50 performs reproduction modeprocessing. The reproduction mode processing is control processing forreproducing a captured image, and an image reproduced on the displayunit 28 can be forwarded or enlarged by operating the operation unit 70.

In step S305, the system control unit 50 performs other processing basedon the determinations in steps S301 and S303. The other processing is,for example, processing of making settings about the digital camera 100and imaging on the setting menu screen.

In step S306, the system control unit 50 determines whether theprocessing is ended. In a case where the processing is ended (YES instep S306), the control process illustrated in the flowchart in FIG. 3ends. In a case where the processing is not ended (NO in step S306), theprocessing returns to step S301. To end the processing refers to, forexample, to turn off the digital camera 100.

FIGS. 4A1 to 4A3 are flowcharts illustrating the imaging mode processingdescribed above as step S302 in FIG. 3 . The processing procedure of theflowcharts is started in a case where it is determined that the imagingmode is set in step S301 in FIG. 3 (YES in step S301 in FIG. 3 ).

In step S401, the system control unit 50 initializes flags and controlvariable numbers.

In step S402, the system control unit 50 displays a LV image 601 on theEVF 29 as illustrated in FIG. 6A, based on a signal acquired via theimage capturing unit 22. A LV image display layer is a layer 705 asillustrated in FIGS. 7A and 7B. Specifically, the LV image 601 isdisplayed on the layer that is lowest in position (lowest in prioritylevel) among those displayed on the EVF 29 according to the presentexample embodiment.

In step S403, the system control unit 50 displays various types ofimaging information (imaging parameter) about the imaging modeprocessing, such as various setting values, the battery level, and thenumber of recording pixels. As illustrated in FIG. 6A, information 602is displayed with the LV image 601 on the EVF 29. Imaging information,such as the information 602 is displayed on a layer higher than the LVimage display layer, as illustrated as a layer 703 in FIG. 7A.

In step S404, the system control unit 50 determines whether a region onwhich AF is executable (hereinafter, “AF executable region”) is smallerthan a region that depends on a main body of the digital camera 100. Ina case where the AF executable region is smaller (YES in step S404), theprocessing proceeds to step S405. In a case where the AF executableregion is not smaller (NO step S404), the processing proceeds to stepS406. The AF executable region is uniquely defined based on performanceof the main body of the digital camera 100, but depending on the type ofthe lens unit 150 mounted on the digital camera 100, the AF executableregion may become smaller than an AF executable region that is based onthe performance of the main body of the digital camera 100. There is acase where an AF executable region smaller than the AF executable regionthat is based on the performance of the main body of the digital camera100 is set in a case where a specific lens, such as a super-telephotosingle focus lens, is mounted because a sufficient amount of light forAF execution may not be obtained with a super-telephoto single focuslens in a region near the periphery of the lens. Thus, in a case wheresuch a lens is mounted as the lens unit 150 on the digital camera 100,an AF executable region smaller than the region that depends on the mainbody of the digital camera 100 is set. Although the user may be aware ofthe AF executable region that depends on the main body of the digitalcamera 100, since the user may use various types of lenses in imaging,when the user changes a lens mounted on the digital camera 100, the usermay not be aware of a range of an AF executable region that depends onthe currently mounted lens. This can lead to a situation where the AFexecutable region that is based on the performance of the digital camera100 is the entire EVF 29 but the AF executable region is reduced as aresult of mounting a specific lens, and this may confuse the user.Unless the user visually recognizes the reduced AF region, the user maynot realize that AF is not executable on a specific area, and the usermay miss an imaging opportunity. In order to prevent the opportunitylosses, the AF executable region determination is performed in stepS404, and in subsequent step S405, the AF executable region isdisplayed. According to the present example embodiment, the AFexecutable region that depends on the main body of the digital camera100 is the entire EVF 29 (vertical-horizontal ratio: 100% x 100%).

In a case where the AF executable region is smaller than the entire EVF29, the line-of-sight pointer displayed in the EVF 29 is moved withinthe AF executable region. Even in a case where the gaze point of theuser is moved to a position outside the AF executable region, theline-of-sight pointer remains within the AF executable region in a stateof staying along a frame 603 indicating the AF executable region.Technically, since the line-of-sight pointer is controlled in such amanner that the circle at the center of the pointer 606 in FIG. 6D comesin contact with the inside of the frame 603, the outside circle of thepointer 606 is displayed beyond the AF executable region. The foregoingcontrol for limiting displaying the line-of-sight pointer outside the AFexecutable region prevents the user from erroneously recognizing thatthe AF frame is movable even to a position outside the AF executableregion.

In step S405, the system control unit 50 displays an indicatorindicating the AF executable region. Since the result of thedetermination in step S404 is “YES”, it is understood that a specificlens that causes the AF executable region to become smaller than the AFexecutable region that depends on the main body of the digital camera100 is mounted. Thus, AF is executable only on the smaller region thanthe entire region of the EVF 29. In order to notify the user of thissituation, the AF executable region is superimposed and displayed overthe LV image. The AF executable region is displayed to be visuallyrecognized by the user as illustrated as the frame 603 in FIG. 6B. Thedisplay form is not limited to the frame display such as the frame 603and can be a display form having a region outside the AF executableregion shaded in gray.

In step S406, the system control unit 50 determines whether the AF areasetting is set to the item other than “entire region”. As describedabove with reference to FIGS. 5A and 5B, the AF area setting is to setwhether to set an entire region or a single point as an area to executeAF. In a case where the AF area setting is set to the item other than“entire region”, i.e., single point (YES in step S406), the processingproceeds to step S407, whereas in a case where the AF area setting isset to “entire region” (NO in step S406), the processing proceeds tostep S408.

In step S407, the system control unit 50 displays the single-point AFframe. A frame 604 in FIG. 6B is an example of the display. Thesingle-point AF frame can be moved to a user-desired position byoperating a direction key of the four-direction key 74.

In step S408, the system control unit 50 determines whether the AFtracking setting is set to “ON”. In a case where the tracking setting isset to “ON” (YES in step S408), the processing proceeds to step S409,whereas in a case where the tracking setting is set to “OFF” (NO in stepS408), the processing proceeds to step S411.

In step S409, the system control unit 50 determines whether a subject isdetected from the LV image acquired via the image capturing unit 22. Ina case where a subject is detected (YES in step S409), the processingproceeds to step S410, whereas in a case where a subject is not detected(NO in step S409), the processing proceeds to step S411. The subjectdetection refers to detection of, for example, a face, a body, or anorgan, such as an eye of a person. A subject is not limited to a person,and an animal, a bird, or a vehicle can be detected.

In step S410, the system control unit 50 displays the subject frame overthe subject detected in step S409. A frame 605 in FIG. 6C is an exampleof the display. Since the result of the determination in step S408 is“YES”, the user may attempt to capture an image of a moving subject.Thus, the subject frame is displayed over the subject detected in the LVimage by the system control unit 50, whereby the subject frame indicatesthe detection of the subject and the user can visually recognize thesubject with ease. According to the present example embodiment, adisplay form of the subject frame includes an angled bracket at each ofthe four corners of the position of the detected subject to surround theregion where the subject is detected.

In step S411, the system control unit 50 determines whether theline-of-sight input function is set to “ON”. In a case where theline-of-sight input function is set to “ON” (YES in step S411), theprocessing proceeds to step S412, whereas in a case where theline-of-sight input function is set to “OFF” (NO in step S411), theprocessing proceeds to step S430. As described above, the line-of-sightinput function is set as desired by the user on the setting menu screen.The line-of-sight input function can be switched between “ON” and “OFF”at a press of a dedicated button for switching the line-of-sight inputfunction or at a press of a button to which the line-of-sight inputfunction can be assigned, without changing to the setting menu screen.

In step S412, the system control unit 50 determines whether aline-of-sight of the user is detected using the line-of-sight detectionblock 160. In a case where a line-of-sight is detected (YES in stepS412), the processing proceeds to step S413, whereas in a case where aline-of-sight is not detected (NO in step S412), the processing proceedsto step S430. According to the present example embodiment, aline-of-sight of the user is not detected in a case where the eye of theuser is not in the proximity of the eyepiece 16.

In step S413, the system control unit 50 determines whether theline-of-sight pointer display setting for the line-of-sight pointerindicating a gaze point is set to “ON”. In a case where theline-of-sight pointer display setting is set to “ON” (YES in step S413),the processing proceeds to step S414, whereas in a case where theline-of-sight pointer display setting is set to “OFF” (NO in step S413),the processing proceeds to step S415.

In step S414, the system control unit 50 displays the line-of-sightpointer at the detected gaze point. The pointer 606 in FIG. 6D is anexample of the display. In FIG. 6D, the frame 605 is displayed over thedetected subject, and the frame 604 indicating a single-point AF frameis displayed. The user is viewing the display position of the pointer606 near the lower right of the LV image 601. FIG. 6E illustrates anexample of a display in a case where the area the user is viewing ischanged to cause the gaze point to move to a position near the displayposition of the frame 605. The line-of-sight pointer 606 is displayed ona display layer lower than the frame 604.

In step S415, the system control unit 50 determines whether aline-of-sight determination operation is performed by the user. In acase where a line-of-sight determination operation is performed (YES instep S415), the processing proceeds to step S416, whereas in a casewhere a line-of-sight determination operation is not performed (NO instep S415), the processing proceeds to step S427. The line-of-sightdetermination operation refers to an operation on the dedicated buttonhaving the line-of-sight determination function or on the button towhich the line-of-sight determination function is assigned. Theline-of-sight determination function refers to a function of determininga gaze point detected at the time of a line-of-sight determinationoperation and invalidating a line-of-sight of the user that is detectedafter the determination and a gaze point calculated based on theline-of-sight detected after the determination. Displaying a framedisplay (item), such as the AF frame, following a detected gaze point onthe EVF 29 may cause the user to feel annoyed. Even in a case where aline-of-sight is unintentionally moved by the user, since the movedline-of-sight is followed by the frame display (item), such as the AFframe, processing such as AF may be executed on an unintended position.Thus, before a line-of-sight determination operation is performed, theframe display is not moved to the gaze point and remains displayed at orover the position/subject determined based on the AF area setting, thetracking setting, and the priority described above, and in a case wherea line-of-sight determination operation is performed, the frame displayis moved to the gaze point and displayed at the gaze point. Since the AFframe is moved from a position to the gaze point and displayed at thegaze point based on the line-of-sight determination operation, even in acase where the AF frame position that is currently displayed on the EVF29 and the user-desired position (gaze point) are at a great distancefrom each other, the AF frame is moved quickly and intuitively. Further,the user intention is accurately read and reflected based on theline-of-sight determination operation by the user. Thus, control such asAF execution is not conducted at a gaze point that is not intended bythe user.

In a case where the frame display before a line-of-sight determinationoperation is disabled (e.g., the AF area setting is set to“entire-region” and the tracking setting is set to “OFF”), display iscontrolled to display the frame display at a determined gaze point in acase where a line-of-sight determination operation is performed.

In step S416, as in step S413, the system control unit 50 determineswhether the line-of-sight pointer display setting is set to “ON”. In acase where the line-of-sight pointer display setting is set to “ON” (YESin step S416), the processing proceeds to step S417, whereas in a casewhere the line-of-sight pointer display setting is set to “OFF” (NO instep S416), the processing proceeds to step S426.

In step S417, as in step S408, the system control unit 50 determineswhether the AF tracking setting is set to “ON”. In a case where the AFtracking setting is set to “ON” (YES in step S417), the processingproceeds to step S418, whereas in a case where the AF tracking settingis set to “OFF” (NO in step S417), the processing proceeds to step S423.

In step S418, as in step S412, the system control unit 50 determineswhether a line-of-sight of the user is detected using the line-of-sightdetection block 160. In a case where a line-of-sight is detected (YES instep S418), the processing proceeds to step S419, whereas in a casewhere a line-of-sight is not detected (NO in step S418), the processingproceeds to step S427.

In step S419, the system control unit 50 determines whether a specificsubject is at the gaze point detected in the LV image in step S418. In acase where a specific subject is at the detected gaze point (YES in stepS419), the processing proceeds to step S420, whereas in a case where aspecific subject is not at the detected gaze point (NO in step S419),the processing proceeds to step S422.

In step S420, the system control unit 50 determines the subjectdetermined as being at the gaze point in step S419 as a tracking targetand executes continues AF on the tracking target subject. At this time,even if the gaze point does not exactly coincide with the position ofthe subject, in a case where the gaze point is detected within apredetermined range from the subject, it is determined that the gazepoint is over the subject, and AF on the tracking target subject iscontinued.

In step S421, a tracking AF frame for the subject determined as atracking target in step S420 is superimposed and displayed over theline-of-sight pointer 606. A frame 607 in FIG. 6F is an example of thedisplay. The tracking AF frame is an AF frame indicating a position ofexecution of continuous AF on the subject that is determined based onthe above-described element condition. The display form of the trackingAF frame includes a double angled bracket at each of the four corners ofthe position of execution of continuous AF to surround the region of thetracking target subject. In a case where the determination result is“YES” in step S415, i.e., the line-of-sight determination operation isperformed by the user, the tracking AF frame is displayed in a displayform different from the frame (subject frame) 605 and the frame (AFframe) 604 that are displayed before the line-of-sight determinationoperation is performed. This enables the user to recognize that thetracking target subject is determined by the line-of-sight determinationoperation performed by the user and to clearly and visually recognizethe subject determined as a tracking target. The subject frame is a meredisplay for notifying the user whether a specific subject (e.g., theface of a person) is in the LV image displayed on the EVF 29 (whether asubject is detected), and in a case where a plurality of subjects isdetected, a plurality of subject frames is displayed. On the contrary, aplurality of tracking AF frames is not displayed in the LV image (onetracking AF frame is displayed), and the display of the tracking AFframe indicates that the subject at the display position of the trackingAF frame is to be focused at a press of the shutter button 61. In otherwords, the subject frame displayed in step S410 and the tracking AFframe displayed in step S421 are different in nature. The subject frameis displayed over a subject recognized as a specific subject in the LVimage by the system control unit 50 but does not indicate that thesubject is an AF execution target candidate. AF is not executed at thedisplay position of the subject frame. On the contrary, the tracking AFframe indicates that the subject is recognized as a specific subject bythe system control unit 50 and is an AF execution target. Further,continuous AF is executed at the display position of the tracking AFframe. Specifically, from the point of view of the importance forrecognition of an AF execution target by the user, the tracking AF frameis more important for the user. Thus, since the tracking AF frame isdisplayed over the gaze point (the position where the line-of-sight isdetected) of the user by a line-of-sight determination operation, thetracking AF frame is quickly moved to a desired position at auser-desired timing.

In step S422, the system control unit 50 displays a multi-point AF frameat the display position of the line-of-sight pointer, i.e., the gazepoint detected based on the line-of-sight of the user. The multi-pointAF frame is an AF frame that is displayed at a position with the highestscore obtained in determination based on the element condition in a casewhere a subject is not detected in the LV image displayed on the EVF 29.The display form of the multi-point AF frame is a small rectangledisplayed at the position with the high obtained score as illustrated asa frame 610 in FIG. 6J. In a case where the multi-point AF frame isdisplayed as in step S422, frames such as the AF frame and the subjectframe are not displayed in the LV image before the line-of-sightdetermination operation is performed in step S415. Specifically, themulti-point AF frame is displayed as a frame display after theline-of-sight determination operation is performed. The multi-point AFframe is displayed on a display layer higher than the display layer ofthe line-of-sight pointer. Since the multi-point AF frame is displayedas described above, the user can visually check that AF is to beexecuted on the subject over which the multi-point AF frame is displayedin a case where an AF execution instruction is issued. Thus, the usercan determine a next operation, such as an operation to issue an AFexecution instruction immediately, an operation to change the positionof the AF frame, or an operation to change the composition or theimaging settings.

In step S423, the system control unit 50 determines whether the AF areasetting is set to “entire region”. In a case where the AF area settingis set to “entire region” (YES in step S423), the processing returns tostep S418, whereas in a case where the AF area setting is not set to“entire region”, i.e., the AF area setting is set to “single point” (NOin step S423), the processing proceeds to step S424.

In step S424, as in steps S412 and S418, the system control unit 50determines whether a line-of-sight of the user is detected using theline-of-sight detection block 160. In a case where a line-of-sight isdetected (YES in step S424), the processing proceeds to step S425,whereas in a case where a line-of-sight is not detected (NO in stepS424), the processing proceeds to step S427.

In step S425, the system control unit 50 moves the single-point AF frameto the display position of the line-of-sight pointer, i.e., the gazepoint. The single-point AF frame is a frame that indicates the positionof a single point to be focused in a case where an AF executioninstruction is issued. The single-point AF frame is displayed in adisplay form such as the display form of the frame 604 in FIG. 6B. In acase where the line-of-sight determination operation is performed by theuser, the single-point AF frame is moved from the display position setbefore the line-of-sight determination operation, to the gaze point.Thus, even in a case where the display position before the line-of-sightdetermination operation is at a great distance from the gaze point, thesingle-point AF frame is quickly moved. Thus, an imaging opportunityloss is reduced and an image focused on a user-desired subject iscaptured and recorded. The single-point AF frame is displayed on adisplay layer higher than the display layer of the line-of-sight pointer606.

In step S426, the system control unit 50 displays a frame display suchas the tracking AF frame, the multi-point AF frame, or the single-pointAF frame at the gaze point, based on the details of the settings of theitems 501 and 502. Since the result of the determination in step S416 is“NO”, the line-of-sight pointer is not displayed on the EVF 29. However,since the results of the determinations in steps S411 and S412 are both“YES”, the line-of-sight of the user is detected using the line-of-sightdetection block 160. As in steps S421, S422, and S425, even in a casewhere the display position of the frame before the line-of-sightdetermination operation is performed is at a great distance from thegaze point, the AF frame is quickly moved. Further, even in a case whereframes are not displayed before the line-of-sight determinationoperation is performed, since the AF frame is quickly displayed at thegaze point, the position of the AF frame is intuitively determined.

In step S427, the system control unit 50 determines whether an AFexecution instruction is issued. In a case where an AF executioninstruction is issued (YES in step S427), the processing proceeds tostep S428, whereas in a case where an AF execution instruction is notissued (NO in step S427), the processing proceeds to step S430. The AFexecution instruction is issued by, for example, pressing the AF-ONbutton 77 or turning on the first shutter switch 62. The AF executioninstruction is not limited to those described above and can be aninstruction by an operation to which the function of issuing aninstruction to execute AF is assigned.

In step S428, the system control unit 50 executes AF on the subject atthe display position of the AF frame displayed over the LV image on theEVF 29. An example of the display is illustrated in FIG. 6H. FIG. 6H isan example of the display in a case where an AF execution instruction isissued in the state illustrated in FIG. 6F (YES in step S427). In a casewhere AF is executed on the subject at the AF frame based on the AFexecution instruction, the display form of the AF frame is changed fromthe display form of the frame 607 in FIG. 6F to a display form of aframe 608 in FIG. 6H based on a focusing result. At this time, in a casewhere the line-of-sight pointer display setting is enabled, the frame608 is displayed on a display layer higher than the display layer of theline-of-sight pointer. Further, in a case where the AF executioninstruction in step S427 is issued at the display position of the frame604 that is the single-point AF frame displayed as illustrated in FIG.6G in step S425, AF is executed on the subject under the frame 604.Then, the display form of the single-point AF frame is changed from thedisplay form of the frame 604 in FIG. 6G to a display form of a frame609 in FIG. 6I based on the focusing result. The display form of the AFframe is changed from the state set before AF execution or based on thefocusing result (focusing is successful and focusing is unsuccessful) tonotify the user of the focus position and the focusing result.Specifically, AF is executed on the subject at the display position ofthe AF frame, and in a case where the focusing is successful, the frame608 in FIG. 6H is displayed in green, whereas in a case where thefocusing is unsuccessful, the frame 608 is displayed in gray. Further,in a case where the focusing is successful, a beeping sound is produced,whereas in a case where the focusing is unsuccessful, no sound isproduced. In a case where an AF execution instruction is issued when theAF area setting is set to “entire region” and the tracking functionsetting is set to “OFF”, the multi-point AF frame is displayed in thedisplay form of the frame 610 in FIG. 6J. The display form of the frame610 is displayed after an AF execution instruction is issued by theuser, and the frame display is not performed before an AF executioninstruction is issued. Specifically, in a case where the AF area settingis set to “entire region” and the tracking function setting is set to“OFF”, the AF frame display is not performed before an AF executioninstruction is issued by the user.

In step S429, the system control unit 50 determines whether the AFexecution instruction determined in step S427 is issued using the firstshutter switch 62. In a case where the determined AF executioninstruction is issued using the first shutter switch 62 (YES in stepS429), the processing proceeds to step S433. Otherwise (NO in stepS429), the processing proceeds to step S430.

In step S430, the system control unit 50 determines whether the firstshutter switch 62 is turned on. In a case where the first shutter switch62 is turned on (YES in step S430), the processing proceeds to stepS431. In a case where the first shutter switch 62 is not turned on (NOin step S430), the processing proceeds to step S437. The state where thefirst shutter switch 62 is turned on refers to the state where theshutter button 61 is half-pressed as described above. Specifically, theuser is expected to perform imaging.

In step S431, the system control unit 50 determines whether a focus modeis set to an AF mode. In a case where the focus mode is set to the AFmode (YES in step S431), the processing proceeds to step S432. Otherwise(in a case where the focus mode is set to a manual focus (MF) mode) (NOin step S431), the processing proceeds to step S434. The AF mode and theMF mode are switched via the setting menu screen or using a switchprovided outside the lens unit 150.

In step S432, as in step S428, the system control unit 50 executes AFprocessing based on the display position of the AF frame.

In step S433, the system control unit 50 performs other imagingpreparation processing, such as AE and AWB.

In step S434, the system control unit 50 determines whether the secondshutter switch 64 is turned on. In a case where the second shutterswitch 64 is turned on, i.e., the shutter button 61 is fully pressed(YES in step S434), the processing proceeds to step S435. Otherwise (NOin step S434), the processing proceeds to step S436.

In step S435, the system control unit 50 performs a series of imagingprocessing up to recording of a captured image as an image file in therecording medium 200.

In step S436, the system control unit 50 determines whether the firstshutter switch 62 is still in the on-state. In a case where the firstshutter switch 62 is still in the on-state (YES in step S436), theprocessing returns to step S431, whereas in a case where the firstshutter switch 62 is no longer in the on-state (NO in step S436), theprocessing proceeds to step S437.

In step S437, the system control unit 50 determines whether the imagingmode is ended. In a case where the imaging mode is ended (YES in stepS437), the control flowchart in FIGS. 4A1 to 4A3 is ended, and theprocessing returns to step S306 in FIG. 3 . On the other hand, in a casewhere the imaging mode is not ended (NO in step S437), the processingreturns to step S403. The imaging mode is ended by, for example, turningoff the digital camera 100 or pressing the menu button 81 to change tothe setting menu screen.

As described above, according to the first example embodiment, in a casewhere an operation can be performed by line-of-sight input, the AF frameis moved to a gaze point and displayed at the gaze point in a case wherea line-of-sight determination operation is performed by the user. SinceAF frame tracking following movement of a line-of-sight of the user isnot performed before the line-of-sight determination operation, theposition of the AF frame is not moved to follow movement of the gazepoint that is not intended by the user. Thus, the user is less likely tofeel annoyed while checking the LV image. Further, even in a case wherethe display position of the AF frame before the line-of-sightdetermination operation is at a great distance from a user-desiredposition, when the user performs a line-of-sight determination operationwith a line-of-sight directed toward a desired position, the AF framecan be quickly moved to the gaze point. Further, since a frame displaysuch as the AF frame is displayed on a display layer higher than thedisplay layer of the line-of-sight pointer in displaying theline-of-sight pointer at the gaze point, the user can clearly recognizea position of the AF frame and also can easily recognize movement of theposition of the AF frame moved to a position toward which theline-of-sight is directed.

Second Example Embodiment

A case where a display order of a frame display that is a display objectsuch as the tracking AF frame, the multi-point AF frame, or thesingle-point AF frame and the line-of-sight pointer indicating a gazepoint is changed before and after the line-of-sight determinationoperation in step S302 (imaging mode processing) in FIG. 3 according toa second example embodiment will be described below. The controlaccording to the second example embodiment will be described below withreference to FIG. 3 and a control flowchart in FIGS. 4B1 to 4B3. FIG. 3is similar to that in the first example embodiment. The flowchart inFIGS. 4B1 to 4B3 is started in the case of the imaging mode processingin the flowchart in FIG. 3 (the case where the result of thedetermination in step S301 is “YES”). Redundant descriptions of steps inthe flowchart in FIGS. 4B1 to 4B3 that are similar to those in FIGS. 4A1to 4A3 according to the first example embodiment are omitted in thesecond example embodiment.

In step S404, in a case where the AF executable region is smaller thanthe region that depends on the main body of the digital camera 100 (YESin step S404), the processing proceeds to step S451. In step S451according to the present example embodiment, the system control unit 50displays the AF executable region (the frame 603 in FIG. 6B) on a layerhigher than the LV image and lower than the imaging information.Specifically, as illustrated in FIG. 7A, the AF executable region isdisplayed on a layer 704 higher than the layer 705 of the LV image andlower than the layer 703 of the imaging information.

In step S452, the system control unit 50 displays the single-point AFframe (the frame 604 in FIG. 6B) on a layer higher than the imaginginformation (imaging parameter). Specifically, as illustrated in FIG.7A, the single-point AF frame is displayed on a layer 702 higher thanthe layer 703 of the imaging information.

In step S453, the system control unit 50 displays the subject frame (theframe 605 in FIG. 6C) over the subject detected in step S409. At thistime, the subject frame is displayed on a layer higher than the imaginginformation. Specifically, as illustrated in FIG. 7A, the subject frameis displayed on the layer 702 higher than the layer 703 of the imaginginformation, similarly to the single-point AF frame.

In step S454, the system control unit 50 displays the line-of-sightpointer indicating the gaze point on a layer higher than the framedisplay, such as the single-point AF frame or the subject frame.Specifically, as illustrated in FIG. 7A, the line-of-sight pointer isdisplayed on a layer 700 higher than the layer 702. Similarly, even in acase where the gaze point is moved as a result of changing a viewingdirection by the user, the line-of-sight pointer is displayed on a layerhigher than the frame display.

Specifically, before the line-of-sight determination operation isperformed by the user in step S415, the line-of-sight pointer isdisplayed on a layer higher than the frame display such as thesingle-point AF frame or the subject frame.

According to the present example embodiment, in addition to setting theline-of-sight input function to “ON” or “OFF”, the user can set also theline-of-sight pointer display setting to “ON” or “OFF”. Thus, in a casewhere the line-of-sight pointer display setting of the item 506 in FIG.5B is enabled, it can be considered that the user wishes a viewingdirection of the user to be detected to view the line-of-sight pointerindicating the gaze point identified based on the detected viewingdirection. Thus, before a line-of-sight determination operation isperformed and the frame display is moved, the display of theline-of-sight pointer is prioritized and displayed on a layer higherthan the frame display. Further, the line-of-sight pointer according tothe present example embodiment is displayed in the display form allowingthe user to check the LV image even in a state where the line-of-sightpointer is superimposed and displayed over the LV image as describedabove. Thus, even in a case where the line-of-sight pointer is displayedon a layer higher than the frame display, the line-of-sight pointer isless likely to completely hide the display position of the frame displayand is less likely to prevent imaging preparation.

In step S456, the tracking AF frame is displayed over the trackingtarget subject determined in step S420. Since this is after theline-of-sight determination operation is performed (YES in step S415),the tracking AF frame is displayed on a layer 712 higher than the layer700 of the line-of-sight pointer as illustrated in FIG. 7B.Specifically, according to the present example embodiment, before theline-of-sight determination operation is performed by the user in stepS415, the frame display is displayed on a layer lower than theline-of-sight pointer (the frame display is displayed on the layer 702in FIG. 7A). On the contrary, after the line-of-sight determinationoperation is performed by the user in step S415, the frame display isdisplayed on a layer higher than the line-of-sight pointer (the framedisplay is displayed on the layer 712 in FIG. 7B).

In a case where the user performs a line-of-sight determinationoperation, it can be considered that the user sufficiently recognizesthe gaze point corresponding to the viewing direction of the user and,furthermore, wishes to move (shift) the frame display to the gaze pointand display the frame display at the gaze point. Further, the displayposition of the frame display such as the single-point AF frame or thesubject frame is a position that is highly important and the user shouldcheck before executing an imaging instruction. Thus, the frame displaythat is displayed and moved based on the line-of-sight determinationoperation is displayed on a layer higher than the line-of-sight pointer.This makes it easy for the user to view the frame display moved by aline-of-sight determination operation and, for example, to recognize asubject that is to be an AF target in a case where an AF executioninstruction is issued.

In step S457, the system control unit 50 moves the multi-point AF frameto the detected gaze point based on the determination result “NO” instep S419. At this time, the multi-point AF frame is displayed on alayer higher than the line-of-sight pointer. Similarly to the trackingAF frame in step S456, the multi-point AF frame is also displayed on alayer higher than the line-of-sight pointer. Specifically, asillustrated in FIG. 7B, the multi-point AF frame is displayed on thelayer 712 higher than the layer 700 of the line-of-sight pointer.

In step S458, the system control unit 50 moves the single-point AF framewhich is displayed on a layer lower than the line-of-sight pointerbefore the line-of-sight determination operation, to the gaze pointindicated by the line-of-sight pointer and superimposes and displays thesingle-point AF frame on a layer higher than the line-of-sight pointer.Specifically, as illustrated in FIG. 7B, the single-point AF frame isdisplayed on the layer 712 higher than the layer 700 of theline-of-sight pointer. Similarly to the tracking AF frame in step S456and the multi-point AF frame in step S457, the frame display of thesingle-point AF frame is moved to the detected gaze point by theline-of-sight determination operation and is displayed on a layer higherthan the line-of-sight pointer. This makes it easy for the user tovisually recognize movement of the single-point AF frame moved to thegaze point by the line-of-sight determination operation and to recognizethe position of the subject on which AF is to be executed in a casewhere an AF execution instruction is issued.

In a case where it is determined that the line-of-sight pointer displaysetting is set to “OFF” in step S416, in step S459, the system controlunit 50 does not display the line-of-sight pointer but displays the AFframe on a layer higher than the imaging information.

As described above, according to the present example embodiment, beforea line-of-sight determination operation is performed by the user, theline-of-sight pointer indicating the gaze point is controlled to besuperimposed and displayed on a layer higher than the AF frame, whereasafter a line-of-sight determination operation is performed, the AF frameis controlled to be superimposed and displayed on a layer higher thanthe line-of-sight pointer. Accordingly, the user can visually recognizethe line-of-sight pointer indicating a gaze point corresponding to aviewing direction of the user with ease before performing aline-of-sight determination operation, and thus the user can predictwhich position the frame display will be moved (shifted) to in a casewhere the user performs a line-of-sight determination operation. On theother hand, after performing a line-of-sight determination operation,the user can easily recognize the position of the frame display moved bythe line-of-sight determination operation, and this makes it easy forthe user to check a target of imaging-related processing such as AF.

The present example embodiment is also applicable to a case where anobject, such as a text file and a folder, is to be selected by aline-of-sight input operation in an electronic device other than adigital camera, such as a PC, a tablet terminal, and a smartphone.

FIGS. 8 and 9A to 9D are diagrams illustrating an example of a displayor a configuration in a case where the present example embodiment isapplied to a PC. FIG. 8 is a diagram illustrating a configuration in acase where the present example embodiment is applied to a PC, and FIGS.9A to 9D illustrate an example of a display screen in a case where thepresent example embodiment is applied to a PC. In FIG. 8 , a touch pad805 corresponding to the touch panel 70 a according to the presentexample embodiment is illustrated. A line-of-sight detection device 810as an independent unit corresponding to the line-of-sight detectionblock 160 for use in a PC is connected to the PC. The eye movement ofthe user is detected using cameras 811 of the line-of-sight detectiondevice 810 and a built-in camera 812 of the PC, and the eye direction(line-of-sight) of the user is detected. A gaze point on a displayscreen is identified based on the detected line-of-sight.

In FIG. 9A, an icon 901 indicating a text file, an icon 902 indicating afolder, an icon 910, a folder name 903, a mouse cursor 904, and aline-of-sight pointer 905 are displayed. The line-of-sight pointer 905indicates a gaze point, and display of the line-of-sight pointer 905 isfor the user to recognize a gaze point identified based on the viewingdirection of the user by the line-of-sight detection device 810. At thistime, the mouse cursor 904 is superimposed and displayed over the icon910, and thus detailed information 906 about the folder indicated by theicon 910 is displayed. At this time, the line-of-sight pointer 905 issuperimposed and displayed over the mouse cursor 904 and the detailedinformation 906. Specifically, the display order of the objects in FIG.9A is the layer 700 of the line-of-sight pointer 905, a layer 701 of themouse cursor 904, and the layer 702 of the detailed information 906 fromthe top as illustrated in FIG. 7A.

A description will be given of a case where the line-of-sight of theuser is moved from the position of the line-of-sight pointer 905 in FIG.9A to a position (on the icon 902) of a line-of-sight pointer 907 inFIG. 9B. After the movement of the line-of-sight, in a case where aline-of-sight determination operation is performed in the stateillustrated in FIG. 9B, the mouse cursor 904 is moved (shifted) to thedisplay position of the line-of-sight pointer 907 as illustrated in FIG.9C. As described above, in a case where the mouse cursor 904 is moved tothe display position of the line-of-sight pointer 907, the mouse cursor904 is moved to the icon 902 indicating the folder, and the detailedinformation 906 about the display object such as the folder or file overwhich the mouse cursor 904 is superimposed and displayed is displayed.Specifically, the detailed information 906 about the object selected bythe mouse cursor 904 as a result of execution of a specific functionthat is executed by moving the mouse cursor 904 in response to theline-of-sight determination operation by the user is displayed. At thistime, the detailed information 906 is superimposed and displayed on alayer higher than the line-of-sight pointer 907. Specifically, thedisplay order of the objects in FIG. 9C is a layer 711 of the mousecursor 904, the layer 712 of the detailed information 906, and the layer700 of the line-of-sight pointer 907 from the top as illustrated in FIG.7B.

In a case where there is not a display object at a position to which themouse cursor 904 is shifted, no detailed object information isdisplayed. The detailed information 906 is displayed after apredetermined time of about one second from when the mouse cursor 904 isshifted to the position of the icon 902 that is the display object.Thus, the detailed information 906 is not displayed in FIG. 9C, which isimmediately after the mouse cursor 904 is shifted by a line-of-sightdetermination operation, whereas the detailed information 906 isdisplayed in FIG. 9D, which is after the predetermined time from whenthe mouse cursor 904 is shifted. The line-of-sight determinationoperation on the PC is not a button operation described above withreference to the digital camera 100 but, for example, a condition that agaze state is continued for a predetermined time, a touch-on operationon the touch pad 805, a click operation on a mouse, or a predeterminedoperation on a keyboard. Accordingly, the user can perform aline-of-sight determination operation more naturally and intuitively.

According to the present example embodiment, detailed information thatis a result of execution of a line-of-sight determination operation isdisplaying of a display layer (e.g., the layer 712 in FIG. 7B) betweenthe mouse cursor 904 and the line-of-sight pointer. Alternatively, thedisplay order of the mouse cursor 904 and the detailed information canbe switched. Further, the mouse cursor 904 and the detailed informationcan be on the same layer. Specifically, the detailed information is tobe displayed on a layer higher than the line-of-sight pointer after aline-of-sight determination operation.

As described above, before a line-of-sight determination operation bythe user, the line-of-sight pointer indicating a gaze point issuperimposed and displayed over detailed information, and thus the usercan visually recognize the gaze point corresponding to a viewingdirection of the user with ease. On the other hand, after aline-of-sight determination operation, detailed information issuperimposed and displayed over the line-of-sight pointer, and thus theuser can check the detailed information with ease without being annoyedby the line-of-sight pointer. Further, after a line-of-sightdetermination operation, the mouse cursor 904 is superimposed anddisplayed over the line-of-sight pointer and detailed information, andthus the user can visually recognize the mouse cursor 904 moved quicklyby a great distance to the gaze point in response to the line-of-sightdetermination operation.

While FIG. 8 illustrates the PC according to another example embodiment,the present example embodiment is implementable also by performing aline-of-sight determination operation using a mouse, a pointing device,or a joystick instead of the touch pad 805 of the PC. Specifically, in acase where a predetermined operation, such as a click operation, a pressoperation, an operation of shaking a device horizontally, or anoperation of drawing a circle is performed, it is determined that theline-of-sight determination operation is performed. The touch pad, themouse, the pointing device, and the joystick do not have to be built inthe PC and can be built in external hardware.

The display form of the line-of-sight pointer is not limited to thosedescribed above. Instead of the form that is a combination of aninternal circle (shaded) and an external circle frame, a form includingan internal circle, a form including an external circle frame, or a formincluding a circle that can be changed in size by the user as desiredcan be employed. For example, a color of the line-of-sight pointer canbe changed based on a staying time of the gaze point. For example, in acase where the staying time of the gaze point is longer than apredetermined time, the color is red, whereas in a case where thestaying time is shorter than the predetermined time, the color is blue.

Means for setting a display form by the user can be provided. Thedisplay order of the display layers can be changed as the display formis changed.

Alternatively, in a case where a line-of-sight determination operationis performed by the user, the frame 604 indicating the single-point AFframe is moved to the position of the line-of-sight pointer 606 and thenthe line-of-sight pointer 606 is hidden. For example, while aline-of-sight determination operation is continued (the first shutterswitch 62 is pressed), i.e., while AF is executed, the line-of-sightpointer 606 is hidden, and when AF execution ends, the line-of-sightpointer 606 is displayed again. The foregoing control makes it easier tovisually recognize the display of the frame 604 indicating thesingle-point AF frame in a situation where the user is likely to wish tocheck the subject at the AF execution position on the LV imageimmediately after the line-of-sight determination operation. The displaycan be controlled to hide the line-of-sight pointer 606, in addition tothe time during AF execution, until a predetermined time set by the userpasses. The frame display such as the AF frame displayed on a layerhigher than the line-of-sight pointer by the determination operation bythe user is reset to the display order before the determinationoperation in a case where the processing proceeds to the mode processingdifferent from the imaging mode processing in step S302 in FIG. 3 .Specifically, the display order is reset in a case where the menu button81 is pressed by the user to proceed to setting menu mode processing orin a case where the digital camera 100 is turned off. A trigger to resetthe display order changed by a determination operation to the displayorder set before the determination operation is not limited to thosedescribed above, and the display order can be controlled to be reset andreturned to the display order set before the determination operation ina case where a predetermined time passes after the determinationoperation.

Further, even during the process of the control flowchart in FIGS. 4A1to 4A3, in a case where the operation unit 70 is operated to move the AFframe or to change various imaging parameters, the functioncorresponding to the operation is executed. For example, in a case wherethe multi-controller 83 is pushed toward any one of the eightdirections, the AF frame is moved based on the operation direction. In acase where the main electronic dial 71 is operated, the function(specifically, shutter speed change) assigned to the main electronicdial 71 is executed.

The present disclosure is also realized by executing the followingprocessing. Specifically, software (program) for realizing the functionsof the above-described example embodiments is supplied to a system or anapparatus via a network or a recording medium, and a computer (or acentral processing unit (CPU) or a micro-processing unit (MPU)) of thesystem or the apparatus reads program codes and executes the readprogram codes. In this case, the program and the recording mediumstoring the program constitute the present disclosure.

The various types of control described above as being performed by thesystem control unit 50 can be performed by a single piece of hardware,or a plurality of pieces of hardware (e.g., a plurality of processors orcircuits) can share the processing to control the entire apparatus.

Further, while the case where the present invention is applied to thedigital camera is described as an example in the above-described exampleembodiments, the disclosed example is not a limiting example, and thepresent invention is applicable to any electric device to which the usercan designate a position by line-of-sight input and operation on anoperation member. Specifically, the present invention is applicable to aPC, a personal digital assistant (PDA), a mobile phone terminal, amobile image viewer, and a head mount display. Further, the presentinvention is also applicable to a digital photo frame, a music player, agame machine, an electronic book reader, a tablet terminal, asmartphone, a projection apparatus, a household appliance that includesa display, and an in-vehicle apparatus that includes a display.

Further, while various example embodiments of the present disclosure aredescribed in detail above, the present invention is not limited to thespecific example embodiments and encompasses various forms within thescope and spirit of the present invention. While the touch panel 70 a isdescribed as an example of a position movement instruction member foruse in combination with line-of-sight input, another operation unit suchas a button or a dial can be employed. Further, while a display positionis indicated by the AF frame, an icon frame or a parameter setting framecan be employed, or an indicator display different from the AF frame,such as the mouse cursor 904, can be employed. While whether to displayor not display the line-of-sight pointer is determined based on usersettings, the determination can be based on whether the line-of-sightinput function is set to “ON” or “OFF” (the item 503 in FIGS. 5A and 5B)or based on whether there is an eye approach to the eyepiece 16.

Further, the present invention is also applicable to a control apparatusthat communicates with an image capturing device (including a networkcamera) via wired or wireless communication and remotely controls theimage capturing device as well as a main body of the image capturingdevice. Examples of an apparatus that remotely controls an imagecapturing device are a smartphone, a tablet PC, and a desktop PC. Thecontrol apparatus remotely controls the image capturing device bynotifying the image capturing device of commands for various operationsand settings based on operations and processing performed on the controlapparatus. Further, a live view image captured by the image capturingdevice can be received via wired or wireless communication and displayedon the control apparatus.

The present invention is also realized by executing the followingprocessing. Specifically, software (program) for realizing the functionsof the above-described example embodiments is supplied to a system or anapparatus via a network or a recording medium, and a computer (or acentral processing unit (CPU) or a micro-processing unit (MPU)) of thesystem or the apparatus reads program codes and executes the readprogram codes. In this case, the program and the recording mediumstoring the program constitute the present invention.

According to the present disclosure, in a case where an operation isperformed by line-of-sight input, a position (designated position)designated by a gaze point is displayed in such a manner that thedesignated position is visually recognized more easily after adetermination operation is performed by a user than before thedetermination operation. This makes it easier to recognize a differencebetween a viewing position of the user and a gaze point before thedetermination of the designated position and also makes it easier tocheck the determined designated position.

Other Embodiments

Various embodiments of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While to example embodiments have been described, it is to be understoodthat the invention is not limited to the disclosed example embodiments.The scope of the following claims is to be accorded the broadestinterpretation so as to encompass all such modifications and equivalentstructures and functions.

1. An electronic device comprising: at least one memory and at least oneprocessor which function as: a detection unit configured to detect agaze point of a user viewing on a display unit; and a control unitconfigured to display an indicator indicating the gaze point detected bythe detection unit on the display unit and, in response to an operationperformed on an operation unit by the user, to display a display item atthe gaze point detected by the detection unit, wherein in a case where asetting to keep displaying the indicator, even in a case where anoperation is performed on the operation unit by the user, is set, thecontrol unit performs control, in response to an operation performed onthe operation unit by the user, to display the display item at the gazepoint detected by the detection unit over the indicator, and wherein ina case where a setting not to display the indicator in response to theoperation is performed on the operation unit, the control unit performscontrol, in response to the operation performed on the operation unit bythe user, not to display the indicator which has been displayed beforethe operation is performed on the operation unit and to display thedisplay item at the gaze point detected by the detection unit.
 2. Theelectronic device according to claim 1, wherein before the operation isperformed on the operation unit by the user, the control unit performscontrol to displays the display item at a position based on apredetermined condition and the indicator on a display layer higher thana display layer of the display item, and in response to the operationperformed on the operation unit by the user, the control unit performscontrol to move and display the display item to the gaze point detectedby the detection unit and to display the indicator on a display layerhigher than a display layer of the display item.
 3. (canceled)
 4. Theelectronic device according to claim 2, wherein before the operation isperformed on the operation unit by the user, the control unit performscontrol to display, the display item at a position based on apredetermined condition and to display the indicator on the displaylayer higher than a display layer of the display item, and in a casewhere the operation is performed on the operation unit by the user, thecontrol unit performs control to display the indicator on a displaylayer lower than a display layer of the display item.
 5. The electronicdevice according to claim 1, wherein before the operation is performedon the operation unit by the user, the control unit performs control tonot display the display item, and in a case where the operation isperformed on the operation unit by the user, the control unit performscontrol to display the display item at a display position of theindicator.
 6. The electronic device according to claim 2, wherein theelectronic device is an image capturing device configured to performimaging processing using an imaging parameter, wherein the control unitfurther performs control to display information about the imagingparameter on the display unit and to display the indicator on a displaylayer higher than a display layer of the information about the imagingparameter.
 7. The electronic device according to claim 1, wherein thedisplay item indicates a position to execute automatic focal pointadjustment.
 8. The electronic device according to claim 1, wherein thecontrol unit performs control to display a second display item indicatesa result of automatic focal point adjustment executed at the positionwhere the display item is displayed.
 9. The electronic device accordingto claim 8, wherein the control unit performs control to display thesecond display item in place of the display item.
 10. The electronicdevice according to claim 1, wherein in a state where the user holds agrip portion of the electronic device with a right hand, the operationunit is disposed at a position which is on an opposite side to a subjectside and on a side closer to a shutter button configured to issue animaging instruction with respect to a central position on a surface onthe opposite side, and which is on a side closer to the grip portion.11. The electronic device according to claim 1, wherein the operation onthe operation unit is a half-press operation to a shutter buttonconfigured to issue an imaging instruction or an operation to instructexecution of autofocus.
 12. The electronic device according to claim 1,further comprising: a finder and an image capturing unit configured tocapture an image of a subject, wherein the display unit is a displayunit inside the finder.
 13. (canceled)
 14. The electronic deviceaccording to claim 1, wherein the display item is a mouse cursor. 15.The electronic device according to claim 1, wherein information about aposition selected by the display item is information about an object ata position where the display item is displayed.
 16. A control method forcontrolling an electronic device, the control method comprising:detecting a gaze point, which is based on line-of-sight of a userviewing, on a display unit; and performing control to display anindicator on the display unit to indicate the gaze point detected by thedetecting on the display unit and, in response to an operation performedon an operation unit by the user, to display a display item at the gazepoint detected by the detecting, wherein in a case where a setting tokeep displaying the indicator, even in a case where an operation isperformed on the operation unit by the user, is set, control isperformed, in response to an operation performed on the operation unitby the user, to display the display item at the gaze point detected bythe detecting over the indicator, and wherein in a case where a settingnot to display the indicator in response to the operation is performedon the operation unit, the electronic device performs control, inresponse to the operation performed on the operation unit by the user,not to display the indicator which has been displayed before theoperation is performed on the operation unit and to display the displayitem at the gaze point detected by the detecting.
 17. A non-transitorycomputer-readable recording medium that stores a program for causing acomputer to execute a control method for an electronic device, thecontrol method comprising, detecting a gaze point, which is based online-of-sight of a user viewing, on a display unit; and performingcontrol to display an indicator on the display unit to indicate the gazepoint detected by the detecting on the display unit and, in response toan operation performed on an operation unit by the user, to display adisplay item at the gaze point detected by the detecting, wherein in acase where a setting to keep displaying the indicator, even in a casewhere an operation is performed on the operation unit by the user, isset, control is performed, in response to an operation performed on theoperation unit by the user, to display the display item at the gazepoint detected by the detecting over the indicator, and wherein in acase where a setting not to display the indicator in response to theoperation is performed on the operation unit, the electronic deviceperforms control, in response to the operation performed on theoperation unit by the user, not to display the indicator which has beendisplayed before the operation is performed on the operation unit and todisplay the display item at the gaze point detected by the detecting.18. The electronic device according to claim 1, further comprising asetting unit configured to set the setting for the display of theindicator in response to an operation by the user, wherein the settingunit is able to set at least a first setting to keep displaying theindicator regardless of the operation performed on the operation unitand a second setting not to display the indicator in response to theoperation performed on the operation unit.
 19. The electronic deviceaccording to claim 18, wherein the setting unit is able to set the firstsetting, the second setting and a third setting not to display theindicator.